Extending Tables with Data from over a Million Websites

This Big Data Track submission demonstrates how the BTC 2014 dataset, Microdata annotations from thousands of websites, as well as millions of HTML tables are used to extend local tables with additional columns. Table extension is a useful operation within a wide range of application scenarios: Imagine you are an analyst having a local table describing companies and you want to extend this table with the headquarter of each company. Or imagine you are a film enthusiast and want to extend a table describing films with attributes like director, genre, and release date of each film. The Mannheim Search Joins Engine automatically performs such table extension operations based on a large data corpus gathered from over a million websites that publish structured data in various formats. Given a local table, the Mannheim Search Joins Engine searches the corpus for additional data describing the entities of the input table. The discovered data are then joined with the local table and their content is consolidated using schema matching and data fusion methods. As result, the user is presented with an extended table and given the opportunity to examine the provenance of the added data. Our experiments show that the Mannheim Search Joins Engine achieves a coverage close to 100% and a precision of around 90% within different application scenarios. 1 Application Example Assume a marketing manager who wants to classify the customers of a company according to different properties of the countries in which the customers are located in order to select those that should be targeted by a marketing campaign. While the data about the customers can be found in the company’s internal data sources, further background information about the customers’ countries is not. Relevant data about countries could for instance include their population, GDP, or human development index. Today, the manager needs to manually search and integrate data about each country using search engines such as Google, access a small set of online databases he knows about, or copyand-paste values from Wikipedia. Manually searching for data is cumbersome and the manager will likely miss a large fraction of the relevant data sources that are available on the Web. The Mannheim Search Joins Engine (MSJ Engine) supports the manager in reaching his goal by automating the data search and data integration tasks, leaving him his core task. 2 O. Lehmberg, D. Ritze, P. Ristoski, H. Paulheim, C. Bizer